Skip to content

feat: Add the file-based override source - #224

Draft
kinyoklion wants to merge 9 commits into
rlamb/overrides-java-override-eventsfrom
rlamb/overrides-java-file-override-source
Draft

kinyoklion wants to merge 9 commits into
rlamb/overrides-java-override-eventsfrom
rlamb/overrides-java-file-override-source

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

This adds the file-based override source that the OVERRIDE specification defines, as FileOverrides.source() in the integrations package, configured through DataSystemBuilder.overrides(...):

LDConfig config = new LDConfig.Builder()
    .dataSystem(Components.dataSystem().defaultMode()
        .overrides(FileOverrides.source().filePaths("/etc/launchdarkly/overrides.json")))
    .build();

The source reads one or more JSON or YAML files in the file data source document format (flags, flagValues, segments) in the configured order. A file whose name ends in .yaml or .yml, in any case, is YAML, and any other file is JSON; a file is read as UTF-8, or in the encoding that a byte order mark names. A flagValues entry becomes a flag that is on and serves the value as its single variation by fallthrough, so it evaluates with the FALLTHROUGH reason kind. Entries keep the versions that the documents specify.

  • duplicateKeysHandling: FAIL (the default) fails the reload when the same key appears in more than one file; IGNORE keeps the entry from the first configured file.
  • changeDetection: POLLING (the default) examines each file's modification time and size on an interval; WATCHING uses file system notifications. The two are alternatives. When the watch service cannot be created at all (for example, the limit on inotify instances is reached), the source logs an error and polls the files instead, at no less than the minimum poll interval.
  • pollInterval: one second by default and at minimum; a smaller value is raised to the minimum with a warning.
  • A configured file that does not exist contributes no overrides and is not an error. Creating it later takes effect, and deleting it removes its overrides.
  • A file that exists but cannot be read or parsed fails that whole reload: the last good overrides stay in effect, the failure is logged, and the source retries after a bounded delay as well as on the next change. An empty JSON file is not a valid document, so it fails the reload too; to remove a file's overrides, delete it or write {}. An empty YAML file has no entries.
  • Files can be rewritten in place. A file whose content is in effect and that a reload finds empty is held back by the reloader from the earlier change: the overrides in effect, and the other files' changes, stay until the new content is written, for at most one retry delay (1 s) after the file was last written while empty and five retry delays from the first hold. FileOverrideSourceImpl passes the loader itself to the reloader, so that the loader learns which content is in effect, and takes the retry delay as a package-private constructor parameter for tests. For a YAML file, replacing its contents in one operation avoids briefly applying a partly written file.
  • The initial load completes during client construction. Every applied change is logged at Info with the overrides in effect and what each file supplied.
  • No file paths, a null change detection mode, a null poll interval, or a file path that is not on the default file system (a zip file system, for example) with WATCHING selected is a configuration error that fails client construction. Polling reads such a path through its own provider. A path listed more than once is logged at Warn and the repeats are ignored; the first occurrence keeps its place in the order.

The public docs say what an application needs: the extension rule and the encoding, that files can be rewritten in place, how to remove a file's overrides, that an empty JSON file is not a valid document, and replacing a YAML file in one operation. The hold's delay and bound stay in the internal comments and in the description of the earlier change. Every public override API, including the builder methods, the poll interval constants and the change detection modes, says that flag overrides are currently experimental and subject to change.

The source is built on OverrideFileLoader, FileDataReloader, FileDataPoller, and FileDataWatcher from the earlier change. The existing file data source keeps its current behavior; this change does not touch its code paths.

This PR depends on the override layer and the reload infrastructure changes and is based on the events branch (rlamb/overrides-java-override-events); retarget it to feat/overrides once that branch merges.

Testing

  • FileOverrideSourceTest: configuration, merging, missing and malformed files, polling and watching reloads, the polling fallback, thread release on close, and at startup an empty YAML file with no entries and an empty JSON file (also with no extension) failing to load.
  • FileOverrideSourceInPlaceRewriteTest, end to end with real in-place rewrites (truncate, pause, write through the same stream): the source never applies a snapshot without the override, for JSON and YAML, watching (300 ms pause) and polling (1.2 s pause, longer than the poll interval); two quick rewrites while watching; the defaults (polling every second, 1 s retry) through the public builder with two rewrites about a second apart; a client with LaunchDarkly data serving a -> b and never the LaunchDarkly value, for JSON and YAML.
  • The in-place rewrite handling was reviewed in eight rounds as fix: Keep the overrides in effect while an override file is rewritten in place #248 (general and adversarial reviews, Bugbot). Mutation check with a clean build per mutant: 57 mutants, each failing at least one test. The override test classes ran 12 times in a row with no failure. The harness flag overrides suite passes against the contract test service at the top of the stack: 38 tests.

SDK-3246

Opened from termag


Note

Overview
Adds FileOverrides.source() so operators can load flag and segment overrides from local JSON/YAML files via DataSystemBuilder.overrides(...), using the same document shape as the file data source.

The builder supports multiple ordered files, duplicate-key FAIL vs IGNORE, POLLING (default, 1s) vs WATCHING change detection, and validation at client build time (paths required, non-default FS + watching rejected). FileOverrideSourceImpl performs a synchronous initial load, pushes full snapshots to the override sink, keeps the last good set on read/parse failures, handles in-place rewrites without dropping overrides, and falls back to polling if the watch service cannot be created.

Docs on OverrideSource and DataSystemBuilder point at FileOverrides. Tests cover configuration, reload behavior, in-place rewrites end-to-end, and client-side allFlagsState semantics for flagValues vs full flags with clientSide.

Reviewed by Cursor Bugbot for commit 43d6c3d. Bugbot is set up for automated code reviews on this repo. Configure here.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from bd3633b to e1dd774 Compare September 28, 2026 21:04
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from bc2e4a9 to 657a598 Compare September 28, 2026 21:04
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from e1dd774 to 752e614 Compare October 1, 2026 23:44
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 657a598 to 5be8948 Compare October 1, 2026 23:44
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from 752e614 to f76f228 Compare October 3, 2026 00:58
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 5be8948 to 9992774 Compare October 3, 2026 00:58
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from f76f228 to b03ecdf Compare October 3, 2026 01:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 9992774 to 1728c29 Compare October 3, 2026 01:43
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from b03ecdf to ed4075d Compare October 13, 2026 21:02
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 1728c29 to c134bcc Compare October 13, 2026 21:02
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from ed4075d to c5cfe53 Compare October 16, 2026 21:18
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from a0f4c55 to 21f7364 Compare October 16, 2026 21:18
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from c5cfe53 to c583704 Compare October 28, 2026 22:19
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 21f7364 to 68fe0d8 Compare October 28, 2026 22:19
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from c583704 to dd7ee83 Compare October 28, 2026 22:39
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 68fe0d8 to 7f0a3eb Compare October 28, 2026 22:39
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from dd7ee83 to 43080b1 Compare October 29, 2026 16:49
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 8ccc74b to af78d3a Compare October 29, 2026 16:49
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from 43080b1 to 0a91418 Compare October 29, 2026 17:01
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from af78d3a to 2ab2fe7 Compare October 29, 2026 17:01
kinyoklion added a commit to launchdarkly/go-server-sdk that referenced this pull request Nov 3, 2026
<!-- TERMAG_PR_BODY_BEGIN -->
## Summary

With `ChangeDetection(Watching)`, a file override source whose file
watcher cannot be created, for example because the limit on inotify
instances is reached, logged "Unable to watch override files" and ran
with no change detection at all. A later edit, creation, or deletion of
an override file never took effect, which the OVERRIDE specification's
Requirement 1.3.7 does not allow ("detecting changes by watching the
files or by polling them, according to platform support").

The source now falls back to polling in that case. It logs at Error
level "Unable to watch override files; polling them instead every
{interval}: {reason}" and polls at the configured interval raised to
`MinimumPollInterval`, because watching mode otherwise ignores the
minimum. Polling mode and a watcher that starts normally are unchanged;
the watcher's own retry for a directory it cannot watch yet is
unchanged.

The same gap was found by Bugbot on the Java implementation
(launchdarkly/java-core#224) and is fixed there the same way.

## Testing


`TestFileSourceWatchingModeFallsBackToPollingWhenTheWatcherCannotBeCreated`
replaces the watch setup with one that fails, then checks that an edit
is applied by polling and that the error is logged. With the poller
creation removed, the test fails. `go test -race ./...` passes (31
packages) and `make lint` reports no issues.
<!-- TERMAG_PR_BODY_END -->

<!-- termag: box=i-0575d1c54853058d7 project=go-server-sdk -->
<sub>Opened from termag</sub>

[SDK-3419](https://launchdarkly.atlassian.net/browse/SDK-3419)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Watching mode** for file override sources no longer leaves change
detection disabled when the OS cannot create a file watcher (for example
inotify limits). The source **falls back to polling** at the configured
`PollInterval`, clamped to **`MinimumPollInterval`**, and logs an error
that includes the poll interval and failure reason.
> 
> Production wiring is unchanged via an injectable **`watchFiles`** hook
(default **`ldfilewatch.WatchOptionalFiles`**) so tests can simulate
watcher creation failure. **Polling mode** and successful watchers are
unaffected.
> 
> A new test verifies that edits are picked up via polling and that the
fallback error message is logged.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
ea03508. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

[SDK-3419]:
https://launchdarkly.atlassian.net/browse/SDK-3419?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
The OVERRIDE specification defines a file-based override source that
reads override entries from one or more local files and reloads them
when the files change. FileOverrides.source() provides it, configured
through DataSystemBuilder.overrides(...).

The source accepts the file data source document format (flags,
flagValues, segments) in JSON or YAML, merges the files in the
configured order with the configured duplicate keys handling (fail by
default, or ignore all but the first), and keeps the versions that the
documents specify. A flagValues entry becomes a flag that is off and
serves the value as its single variation. Change detection is polling
(the default, one second by default and at minimum) or watching. A
configured file that does not exist contributes no overrides. A file
that exists but cannot be read or parsed fails that reload, keeps the
last good overrides, logs the failure, and is retried. The initial load
completes during client construction. Every applied change is logged at
Info with the overrides in effect and what each file supplied. Missing
file paths, a null change detection mode, or a null poll interval fail
client construction.

Flag overrides are currently experimental and subject to change.
The poller logs a change callback that throws, so it needs the source's logger. This compiles once the branch includes the poller constructor that takes a logger.
…ntries as duplicate keys

A path listed more than once would make every entry of that file a duplicate of itself: under the
default fail handling the load failed, and under ignore the first listing won by accident. The
builder now drops the repeats, keeps the first occurrence in its place in the order, and logs a
warning once at build time. This is the behavior the Node SDK has.
The watcher uses the default file system's watch service, which cannot
watch a path of another provider such as a zip file system. Registering
one threw an unchecked exception from start, so a configuration error
surfaced as a failed start. The builder now rejects such a path when
watching is selected, so the error is reported when the client is
created. Polling reads such a path through its own provider and still
accepts it.
When watching was selected and the watch service could not be created,
for example because the limit on inotify instances was reached, the
source logged the failure and ran with no change detection, so a later
edit never took effect. It now polls the files instead, at no less than
the minimum poll interval, and the log says so.
… SDKs

A flagValues entry becomes a flag without client-side availability, so
allFlagsState with CLIENT_SIDE_ONLY leaves it out even when the flag it
overrides is client-side. A full definition with clientSide is the way
to override such a flag. A test pins both.
… in place

The file-based override source now uses the reloader's handling of a file
that is emptied during an in-place rewrite: it passes the loader itself to
the reloader, so that the loader learns which content is in effect, and
takes the retry delay as a parameter for tests.

The docs say how the format of a file is chosen (.yaml and .yml are YAML,
any other file is JSON), how a file is decoded (UTF-8, or the encoding that
a byte order mark names), that files can be rewritten in place, how to
remove a file's overrides, that an empty JSON file is not a valid document,
and that replacing a YAML file in one operation avoids briefly applying a
partly written file. Every public override API says that flag overrides
are experimental. End-to-end tests rewrite override files in place, with
watching and polling, and check that the override is never missing.
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-override-events branch from 0a91418 to 973c806 Compare November 3, 2026 23:55
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-java-file-override-source branch from 2ab2fe7 to 43d6c3d Compare November 3, 2026 23:55
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 43d6c3d. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant